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ABSTRACT 


This  paper  reports  the  results  of  the  second  phase  of  a  two-phase  project 
to  develop  user  feedback  mechanisms  for  selected  Defense  Technical  Information 
Center  (UTIC)  products/services.  Self-administered  questionnaires  are  proposed 
for  seven  products/services.  In  addition,  a  method  (in  grid  format)  is  presented 
which  DTIC  personnel  can  use  to  analyze  conference  minutes  so  as  to  identify  and 
categorize  user  concerns.  Used  over  time,  these  questionnaires  and  the  grid 
would  produce  data  which  could  be  used  to  identify  agency  strengths  and 
weaknesses  and  to  project  trends. 
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This  paper  reports  the  results  of  the  second  phase  of  a  two-phase  project 
to  develop  a  set  of  user  feedback  mechanisms  for  selected  Defense  Technical 
Information  Center  (DTIC)  products  and  services.  The  results  of  the  earlier 
effort  were  reported  in  "User  Feedback  Mechanisms  for  Defense  Technical 
Information  Center  Products  and  Services,  Phase  I,"  dated  January, 

198J.  During  this  second  phase  self-administered  questionnaires  were  developed 
for  users  of  Demand  Bibliographies,  Current  Awareness  Bibliographies  (CAB)  and 
Recurring  Management  Information  System  Reports;  for  participants  in  the 
Automatic  Distribution  of  Documents  (ADD)  program,  for  students  in  online 
training  classes,  for  conference  attendees  and  for  visitors  to  the  DTIC  facility. 
These  products  and  services  were  selected  because  they  are  central  to  the 
accomplishment  of  the  DTIC  mission  and  because  they  are  entitles  which  users  are 
capable  of  evaluating  realistically.  In  addition  to  these  instruments  to  collect 
direct  feedback,  a  method,  in  grid  format,  is  presented  which  DTIC  personnel  can 
use  to  analyze  conference  minutes  so  as  to  identify  and  categorize  user  concerns. 
Used  over  time  these  questionnaires  and  the  grid  will  produce  data  which  can  be 
used  to  identify  agency  strengths  and  weaknesses  and  to  project  trends. 

OVERVIEW  OF  APPROACH 

Three  preliminary  steps  were  taken  during  Phase  I  of  this  project.  First,  a 
literature  search  was  made  to  identify  previous  efforts  to  develop  feedback 
mechanisms  for  DTIC  products  and  services.  Second,  similar  agencies,  e.g.  the 
National  Technical  Information  Service  and  the  National  Aeronautics  and  Space 
Administration  Technical  Information  Center,  were  queried  to  identify  feedback 
mechanisms  they  use  for  products  and  services  similar  to  those  produced  by  DTIC. 
Third,  workers  in  various  DTIC  programs  were  canvassed  to  identify  existing 
formal  and  informal  efforts. 

During  Phase  II  those  feedback  mechanisms  which  were  located  were  examined 
and  evaluated  for  procedures  which  could  be  adapted  for  this  project.  The  writer 
then  met  with  work  groups  in  the  various  DTIC  programs  and  explained  the  goals  of 
this  project.  Workers  were  asked  to  describe  the  goals  their  programs  were 
intended  to  achieve.  They  were  then  asked  to  think  of  ways  in  which  user 
feedback  could  help  them  achieve  those  goals.  Based  on  the  information  gained 
through  the  literature  search,  contributions  from  similar  agenlces,  existing  DTIC 
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forms  and  most  importantly,  input  from  the  program  workers,  draft 
questionnaires  were  developed  by  the  writer. 

When  a  draft  questionnaire  acceptable  to  both  the  writer  and  the  program 
personnel  had  evolved  for  all  the  programs,  the  drafts  were  then  pre-tested. 

Eight  professional  librarians  who  serve  significant  research  and  development 
populations  cooperated  in  this  pre-test  effort.  Appendix  A  lists  the 
participants  in  the  pre-test. 

Pre-testing  a  questionnaire  gives  the  researcher  an  opportunity  to  see  how 
well  it  works  and  what  changes  are  necessary  before  the  start  of  a  full-scale 
study.  This  preliminary  step  can  significantly  improve  the  survey  instrument  and 
greatly  enhance  the  quality  of  the  information  obtained.  During  the  pre-test  the 
researcher  can  identify  and  solve  unforeseen  problems  in  the  administration  of 
the  questionnaire,  such  as  the  wording  and  sequence  of  questions  or  Inadequate 
instructions  for  filling  out  answers  to  questions.  It  may  also  Indicate  that 
certain  additional  questions  are  needed  or  that  certain  ones  should  be 
eliminated. 

In  general,  the  pre-test  should  be  in  the  form  of  personal  Interviews,  as 
was  done  in  this  case.  An  important  part  of  the  pre-test  interview  is  discussion 
of  tlie  questions  with  the  respondents  after  they  have  answered  them.  The 
respondent  may  be  queried  as  to  what  the  questions  meant  to  him  or  her,  what 
difficulties  were  experienced  in  replying,  what  further  ideas  he  or  she  had  in 
mind  which  were  not  brought  out  by  the  question,  and  how  the  respondent  would 
have  asked  the  question.  There  was  a  consistent  effort  to  follow  these 
guidelines  in  pre-testing  these  questionnaires. 

The  pre-test  participant  input  was  evaluated  by  the  writer.  Most  of  the 
suggested  changes  were  incorporated  into  the  draft  questionnaires.  The 
pre-tested  forms  were  then  taken  back  tv  program  workers  for  further 
consultations.  The  vast  majority  of  the  changes  were  readily  accepted  by  the 
program  workers. 

OVERVIEW  OF  DATA  ANALYSIS 

In  analyzing  the  user  input,  the  researcher  will  take  advantage  of  the 
speed,  convenience  and  flexibility  offered  by  the  Statistical  Package  for  the 


Social  Sciences  (SPSS),  available  at  OTIC  on  the  Univac  1100/60.  SPSS  is  a 
packaged  computer  program  specifically  designed  to  compute  those  statistics 
typically  used  by  social  scientists.  Such  statistics  Include  single-variable 
descriptive  statistics,  for  example,  median,  mean,  mode,  range,  variance, 
standard  deviation,  and  skewness.  SPSS  will  also  provide  the  capability  to  do 
crosstabulations  and  related  measures  of  association.  SPSS  output  is  available 
in  tabular,  histogram  and  scattergram  form. 

The  researcher  will  provide  OTIC  management  with  reports  based  on  the  SPSS 
output.  Having  such  reports  should  provide  DTIC  management  with  practical 
planning  information.  They  would  be  advised  of  the  specific  characteristics 
users  want  in  products  and  services.  They  would  be  alerted  to  areas  where 
problems  exist  or  are  developing,  such  as  areas  where  user  training  is 
particularly  needed.  Similarly,  reviewing  the  results  of  surveys  over  a  number 
of  years  would  enable  planners  to  more  accurately  project  long-range  trends  in 
user  needs. 

BRIEFING  AND  TOUR  QUESTIONNAIRE 

DTIC  welcomes  visitors  with  a  monthly  briefing  and  tour  of  the  facility. 

The  vast  majority  of  these  visitors  are  users  or  potential  users  of  DTIC  products 
and  services.  A  DTIC  spokesperson  summarizes  the  history  of  the  agency  and 
provides  an  overview  of  its  mission  and  programs.  Visitors  are  then  shown  a 
video  slide/tape  presentation,  "Information:  A  Basic  Research  and  Development 
Tool,"  which  further  clarifies  the  agency's  role  in  DoD  research  and 
development.  Following  the  slide/tape  presentation,  visitors  tour  the  facility. 
Emphasis  is  placed  on  those  work  stations  which  experience  has  shown  to  be  of  the 
greatest  interest  to  visitors.  For  example,  there  is  a  demonstration  of  on-line 
searching.  The  briefing  and  tour  together  typically  require  one  and  one-half 
hours.  From  1  May  1982  to  1  May  1983,  130  visitors  received  the  briefing  and 
tour. 

The  evaluation  form  (see  Figure  1)  for  this  service  was  designed  in 
consultation  with  Thomas  Lahr,  Office  of  User  Services,  the  present  DTIC 
briefer.  In  Part  I  of  the  questionnaire  respondents  are  asked  to  identify  the 
type  of  organization  with  which  they  are  associated.  Part  II  is  designed  to 
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Figure  1 

DEFENSE  TECHNICAL  INFORMATION  CENTER 

_ BRIEFING  AND  TOUR _ 

You  have  just  completed  a  briefing  and  tour  to  familiarize  you  with  DTIC.  We  are 
interested  in  your  comments  so  that  we  may  improve  the  experience  for  future  visitors.  We 

would  appreciate  your  frankness  and  cooperation  in  completing  this  evaluation. _ 

PART  I _ VISITOR  INFORMATION _ 

1.  Organization 

DoD  Other  US  Government  Contractor  Other 


PART  II _ COMMENTS  ON  BRIEFING/TOUR 

2.  Organization  of  Briefing/Tour 

_  Excellent  _  Satisfactory  _  Needs  improvement 


3.  Duration  of  Briefing/Tour 

_  Too  Short  _  Appropriate  _  Too  Long 


4.  Visual  appeal  of  the  slide/tape  presentation 

_  Excellent  _  Satisfactory  _  Needs  improvement 


5.  Information  conveyed  by  the  slide/tape  presentation 

_  Clearly  useful  _  Useful  _  Marginally  useful 


6.  Comments 


PART  III _ COMMENTS  ON  BRIEFER _ 

7.  Briefer's  knowledge  of  DTIC  operations  and  services 

_  Comprehensive  _  Adequate  _  Needs  Improvement 


B.  Did  the  Briefer  seem  to  be  open  to  questions?  Yes  No 


9.  Did  the  Briefer  answer  your  questions  or  offer  to  research  questions  for  which  he/she 
had  no  immediate  answer?  Yes  No 


elicit  information  on  whether  the  program  could  more  effectively  be  presented 
in  some  different  sequence  and  whether  visitors  believe  that  a  balance  was 
achieved  between  their  need  for  information  about  the  agency  and  the  amount  of 
time  that  they  were  required  to  expend  to  meet  that  need.  Questions  on  the 
visual  appeal  and  information  conveying  capacity  of  the  slide  tape  presentation 
are  designed  to  alert  DTIC  to  any  need  to  seek  a  new  audio-visual  package. 

Part  III,  "Comments  on  Briefer,"  is  designed  to  give  managers  a  measure  of 
the  success  of  one  of  the  most  critical  elements  in  the  agency’s  public  relations 
program.  Of  even  greater  importance,  answers  to  these  questions  will  provide  the 
briefer  himself  or  herself  with  information  as  to  which  aspects  of  his  or  her 
performance  are  effective  and  which  require  greater  effort  or  a  re-examination  of 
approach. 

The  structured  questions  on  this  form  are  designed  so  as  to  be  readily 
analyzable  and  to  provide  data  which  will  be  comparable  from  one  year  to  the  next 
so  that  trends  can  be  Identifed.  In  contrast,  the  open-ended  "Comments"  section 
(f/b)  and  the  "Suggestions"  section  (#14)  provide  for  more  spontaneous  responses 
which,  if  the  respondent  so  chooses,  can  be  related  to  specific  events  in  a 
particular  briefing  session. 

UTIC  personnel  will  supply  the  number  of  visitors  called  for  at  the  end  of 
the  form  so  chat  size  of  the  visitor  group  can  be  related  to  reports  of 
successful  or  unsuccessful  briefing  experiences.  The  date  of  the  briefing,  also 
supplied  by  UTIC,  will  be  sufficient  to  identify  the  briefer.  This  form  should 
be  distributed  to  all  UTIC  visitors. 

Most  of  tlie  participants  in  the  pre-test  found  this  questionnaire  to  be 
straightforward.  The  purposes  of  the  questions  seemed  clear  to  them.  However, 
several  were  unsure  whether  or  not  they  were  expected  to  supply  the  number  of 
visitors  in  the  tour.  To  overcome  this  problem,  the  final  version  of  the  form 
should  have  a  heavy  line  dividing  the  "DTIC  Use  Only"  section  from  the  previous 
section. 

Another  problem  became  evident  on  (Questions  2  and  9.  On  the  test  version  of 
the  form  these  two  questions  had  check-block  series  labeled  "excellent,  good, 
average  and  needs  improvement,"  Several  respondents  said  that  for  them  "good" 
and  "average"  have  Che  same  meaning.  As  a  consequence,  "satisfactory"  was 
substituted  for  "good"  and  "average." 


One  respondent  suggested  that  Part  II  should  have  a  comments  section.  Space 
limitations,  however,  make  it  impractical  to  Implement  this  suggestion.  Another 
respondent  stated  that  the  "Comments  on  Briefer”  section  is  particularly  needed 
since  UTIC  presenters  at  public  gatherings  frequently  make  a  poor  showing. 

CONFERENCE  EFFECTIVENESS  QUESTIONNAIRE 

As  part  of  its  ongoing  user  liaison  and  training  effort  OTIC  annually 
sponsors  seven  user  conferences.  Six  regional  conferences  are  dispersed  around 
the  country  so  as  to  minimize  travel  time  and  expense  for  users.  One  national 
conference  is  held  near  the  agency's  main  facility.  This  national  conference 
permits  maximum  interaction  between  greater  numbers  of  agency  personnel  and  many 
of  the  users  of  OTIC  products  and  services.  It  also  provides  an  opportunity  for 
the  Administrator  and  his  staff,  on  the  one  hand,  to  summarize  the  year's 
problems  and  accomplishments  and  to  react  as  a  group  to  questions  of  pervasive 
user  interest.  Un  the  other  hand,  the  user  advisory  council  provides  a  review 
and  judgment  from  the  user  viewpoint  and  a  summation  of  tasks  accomplished  versus 
new  or  unfinished  requirements.  The  six  regional  conferences  scheduled  for  1983 
were  lield  between  14  March  and  6  May  with  317  user-participants  present.  The 
most  recent  national  conference,  October,  1982,  in  Alexandria,  Virginia, 
attracted  270  users. ^  These  conferences  give  attendees  an  opportunity  to 
receive  training  in  the  use  of  agency  products  and  services  and  to  benefit  from 
interaction  with  peers  from  other  information  handling  agencies.  In  addition, 
the  conferences  give  agency  personnel  an  opportunity  to  obtain  both  informal  and 
torr.ial  feedback  from  users  and  are  therefore  critical  to  DTIC  efforts  to  improve 
its  products  and  services. 

This  questionnaire  (see  Figure  2)  was  designed  to  facilitate  gathering  of 
user  feedback  at  conferences.  Roberta  Cohen,  Office  of  Information  Systems  and 
Technology,  developed  a  questionnaire  which  was  distributed  at  the  six  1983 
regiot  jl  conferences.'^  That  questionnaire  and  the  user  responses  to  it  were 
studied  and  offered  a  number  of  useful  insights  for  development  of  this 
questionnaire.  Paul  Klinefelter,  Judy  Fickeral  and  Thomas  Lahr,  Office  of  User 
Services,  were  consulted  as  part  of  the  development  process.  They  offered  a 
uumher  of  siiggestions  whicli  were  also  incorporated.^ 

Part  1  of  tfk?  questionnaire  requests  considerable  information  about  the 


CONFERENCE  EFFECTIVENESS  SURVEY 


6.  Professional  information  meetings  you  have  attended  or  expect  to  attend  this  year, 
(check  all  that  apply) 

_  DTIC  Regional  Conference( s)  _  Online  YR?  (Sponsored  by  Online  Review) 

_  DTIC  Annual  Conference  _  Federal  DP  EXPO 

_  AS IS  _  Other  _ 

_  SLA  _ . 


7.  Approximately  how  far  did  your  travel  to  come  to  this  conference? 
_  Under  50  miles  _  50  -  100  miles  _  100  plus  miles 


PART  II  CONFERENCE  EVALUATION 


What  IS  your  overall  evaluation  of  this  conference  on  the  following  scale?  (check  one) 
very  little  of  no 

useful  useful  use  use 


9.  Which  session  or  aspect  of  the  conference  did  you  find  MOST  useful?  (Use  other  side 
if  needed) 


10.  Which  session  or  aspect  of  t’  conference  did  you  find  LEAST  useful?  (Use  other  side 
if  needed) 


PART  III  FUTURE  CONFERENCES 


11.  DTIC  wants  you  to  help  plan  the  agenda  for  future  conferences.  What  problem  areas 
should  be  addressed?  (Use  other  side  if  needed). 


12.  EVALUATOR  NAME  AND  ORGANIZATION  (OPTIONAL) 


DTIC  USE  ONLY 
Conference  ID 


users  themselves.  This  Information  is  important  to  have  so  that  training 
sessions  can  be  tailored  to  the  actual  needs  of  the  persons  who  attend  DTIC 
conferences.  Questions  1  through  5  are  directed  particularly  toward  that  need. 
Questions  6  and  7  are  intended  to  measure  limits  on  users'  opportunities  to  gain 
first-hand  information  about  advances  in  information  handling  and  to  interact 
with  peers. 

Part  II,  Question  8,  is  intended  to  be  a  gross  measure  only.  However,  it 
should  be  informative  when  results  are  summarized  and  compared  over  several 
years.  The  remaining  three  questions  are  open-ended  and  are  intended  to  gather 
information  for  planning  the  specifics  of  subsequent  conferences.  Questions  9 
and  iO  focus  on  the  perceived  strengths  and  weaknesses  of  the  conference  the 
respondent  is  attending.  Question  11  seeks  to  identify  the  concerns  that  users 
consider  serious,  which  they  believe  will  be  of  a  persistent  character  and  which 
they  believe  can  be  profitably  addressed  at  DTIC  conferences. 

Question  2  of  the  first  version  of  this  questionnaire  listed  only 
"Librarian"  and  "Library  Technician"  as  position  titles  for  persons  working  in 
libraries.  In  response  to  pre-test  participant  suggestions,  a  third  category, 
"Library  Director,"  was  added.  Several  of  the  pre-test  participants  said  that  in 
their  minds  the  term  "librarian"  applies  only  to  persons  offering  direct  service 
to  users  and  that  some  other  designation  is  required  to  identify  a  library 
administrator. 

One  respondent  objected  to  Question  8,  maintaining  that  it  is  too  general  to 
provide  any  useful  information.  This  same  respondent  predicted  that  Questions  9 
and  10  would  provide  the  specific  information  DTIC  needs. 

Another  respondent  was  enthusiastic  about  Question  11,  saying  that  it  is 
highly  advisable  to  seek  this  sort  of  planning  information  at  the  end  of  a 
conference  when  the  events  and  experiences  of  the  conference  are  still  fresh  in 
the  minds  of  the  attendees.  Still  another  said  that  by  its  "vagueness"  this 
question  suggests  that  DTIC  is  unwilling  to  do  the  work  of  planning  its  own 
conferences.  This  participant  held  that  it  would  be  more  effective  to  provide  a 
list  of  possible  agenda  topics  and  ask  the  respondents  to  prioritize  those 
topics.  This  suggestion  may  have  considerable  merit.  However,  it  was  not 
incorporated  into  the  questionnaire,  because  a  list  of  specifics  would  tend  to 
date  a  form  which,  it  is  hoped,  can  be  used  over  several  years. 

Still  another  respondent  felt  that  there  should  be  a  "general  comments" 
section.  This  suggestion  was  not  incorporated  into  the  form  because  of  space 
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limitations  and  because  it  is  believed  that  Questions  9,  10  and  11  provide 
sufficient  opportunity  for  open-ended  response. 

These  questionnaires  should  be  given  to  attendees  when  they  register  both 
for  regional  and  national  conferences.  Potential  respondents  should  be  requested 
to  return  the  questionnaires  at  the  end  of  the  conference.  Those  who  fail  to 
return  the  questionnaire  could  differ  significantly  from  those  who  do  return  it. 
These  persons  whose  response  is  missed  may,  thereby,  introduce  an  element  of 
non-response  bias  into  the  results.  However,  previous  conference  attendees  have 
shown  themselves  to  be,  as  a  group,  interested  in  cooperating  with  efforts  to 
obtain  their  input.  For  that  reason,  it  is  expected  that  the  non-response  bias 
will  not  be  sufficient  to  invalidate  the  data. 

ON-LINE  TRAINING  PROGRAM  QUESTIONNAIRE 

UTIC  offers  an  on-going  program  to  train  representatives  of  proposed  or 
operational  sites  in  the  use  of  remote  terminals  linked  to  the  Defense  RDT&E 
On-Line  System  (UROLS).  At  the  end  of  FY  1982  the  on-line  remote  terminal  system 
included  541  user  organizations.^ 

Basic  and  refresher  courses  for  beginning  and  intermediate  level  students 
stress  the  principles  operators  must  master  for  successful  retrieval  of 
information  from  OROLS.  Workshops  give  more  advanced  students  an  opportunity  to 
further  refine  their  retrieval  skills.  Another  series  of  classes  teaches  data 
input  technique  to  representatives  of  Information  Analysis  Centers, 
representatives  of  sites  participating  in  the  Shared  Bibliographic  Input  Network 
and  to  personnel  from  sites  which  input  their  project  data  directly  into  the  Work 
Unit  Information  System. 

During  FY  1982,  141  students  were  trained  in  retrieval  at  the  DTIC  site. 
Other  retrieval  training  sessions,  attended  by  83  students,  were  presented  at 
nine  locations  around  the  United  States.  Training  for  input  is  centralized  at 
DTIC  where  35  students  were  trained  in  FY  1982.^ 

This  form  (see  Figure  3)  was  designed  in  consultation  with  James  DePersis, 
Marie  Clark,  and  Jane  Hatton,  Directorate  of  Data  Base  Services.®  These  DTIC 
trainers  were  divided  as  to  the  usefulness  of  any  training  evaluation 
questionnaire.  One  felt  that  students  would  not  provide  objecti”e  critiques  of 


you  have  just  completed  a  training  program  to  learn  to  operate  the  Defense  RDT&E  On-line 


system.  DTIC  is  interested  in  your  comments  concerning  the  course  content,  format  and 
presentation  so  that  future  instruction  can  meet  your  needs.  We  would  appreciate  your 

frankness  and  cooperation  in  completing  this  evaluation. _ 

Part  I _ PRELIMINARY _ 

A.  TRAINING  RECEIVED _  _ 


1 .  Type  of  training 

2.  Level  of  training 

Retrieval 

Dedicated  Dial-up 

Basic 

Refresher 

_  Input  _ 

SBI 

Workshop 

3.  Data  Base(s) 

4.  Length  of  course 

TR  file  (1473) 

PP  file  (1643) 

2  days 

5  days 

_  WU  file  (1498) 

_  IR&D  (271) 

3  days 

10  days 

B.  YOUR  CURRENT  POSITION 
1 .  Organization 


_  DoD  _  Other  US  Government  _  Contractor  _  lAC  _  Other 

_ 

I  2 .  Your  position 

_  Librarian 

Libiary  Technician 
Tech.  Info.  Specialist 

C.  OTHER  ON-LINE  SYSTEMS _ 

|l.  Available  at  your  site  (check  all  that  apply) 

DoE/RECON  NASA/RECON  DIALOG  ORBIT  BRS  Other 


Scientist/Engineer  _  Clerk/Steno 

programmer  _  Other  (Specify) 

Administrative  Asst. 


2.  tifhich  ones  can  you  operate?  (check  all  that  apply) 

DoE/RECON  NASA/RECON  DIALOG  ORBIT 


Other 


13.  flow  did  this  knowledge  affect  your  learning  the  DTIC  system? 
1  Helped  Hindered  Neither 


IPART  II 


..  Training  materials 


2.  Reference  tools 


.  Visual  aids 


.  Trainin 


COMMENTS  ON  MATERIALS,  FACILITY  AND  EQUIPMENT 


Excellent  |  Good 


Ease  of  equipment  operation 


.  Printer 

1 

.  Sound  Page  Recorder 

1 

.  Tape  Casette 


Figure  3-Contlnued 


PART 

Ill 

COMMENTS  ON  LEARNING 

EXPERIENCE 

A. 

Data  Base  Coverage 

Excellent 

Good 

Average 

Needs 

Improvement 

1  . 

TR  File  ( 1473) 

2. 

WUIS  (1498) 

3. 

PP  File  (1643) 

4. 

IR&D  (271) 

B. 

Subject  content  of  the 

course 

1 . 

Theory 

Too  Much 

Balanced 

Too  Little 

2. 

Practical 

Too  Much 

Balanced 

— 

Too  Little 

C. 

Hands-on  Practice 

_  Helpful 

_  Not  Needed 

Comments 

I 

I 

D. 

! 

Security  Coverage 

Excellent 

- 

Adequate 

- 

Needs 

Improvement 

Comments 


E.  Length  of  Course 

1.  Leave  as  is  Longer  (number  of  days? _ )  _  Shorter  (number  of  days? _ ) 


I  2.  Suggested  additional  subjects 


I 


i  3. 

I 

What  would  you  omit: 

i 

! 

i 

Additional  training  you 

_  Yes  2. 

_  No 

believe  you  will 

_  Refresher 

_  Workshop 

need 

3.  _ 

in  3 
in  6 

months 
-  12  months 

iPART 

IV 

EVALUATION 

OF  INSTRUCTOR 

1  Instructor's  Name 

Date 

Organization  of  subject 

matter 

Excellent 

Good 

Needs 

improvement 

2. 

Delivery  of  subject  matter 

Excellent 

Good 

Needs 

improvement 

3. 

Attitude  of 

instructor 

(check  all 
Friendly 

that  apply) 

_  Distant 

Helpful 

Not  helpful 

4. 

Atmosphere 

established 

Formal 

Informal 

Comments 


PART 

V 

OVERALL 

EVALUATION  OF  COURSE 

1. 

— 

Excellent 

Very  Good 

_  Average 

Needs  improvement 

2. 

Would 

you  recommend 

this  course  to 

others?  Yes 

_  No 

^Comments 


•  «  F  . 
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the  course,  because  the  questionnaires  would  be  administered  by  the  trainer  whose 
course  they  were  being  asked  to  evaluate.  On  the  other  hand,  another  Instructor 
maintained  that  previous  training  evaluation  questionnaires  had  provided  useful 
information  and  that  this  questionnaire  could,  in  fact,  contribute  to  an  improved 
training  program.  Since  the  objection  to  the  questionnaire  was  focused  on  the 
manner  of  its  administration  rather  than  on  the  concept  of  formal  student 
feedback  per  se,  it  is  reasonable  to  develop  and  implement  the  questionnaire.  If 
administration  of  the  questionnaire  by  the  instructor  does,  in  fact,  produce 
little  or  no  negative  feedback,  then  other  DTIC  personnel  could  administer  the 
questionnaire.  Both  of  these  approaches  are  used  in  administering  student 
evaluations  of  university  courses.  Experience  would  indicate  which  approach 
would  be  most  efficacious  for  DTIC. 

This  questionnaire  is  suitable  for  use  with  both  retrieval  and  input 
students.  It  should  be  distributed  at  the  final  session  of  a  training  series. 
Parc  I,  Section  A,  requests  information  required  to  relate  course  evaluations  to 
the  particular  type  of  training  received.  Persons  attending  these  classes  vary 
widely  in  training  and  experience  and  in  their  current  roles  in  their 
organizations.  Section  B  seeks  information  so  that  training  evaluations  can  be 
related  to  these  factors.  Similarly,  Section  C  aims  to  relate  previous  exposure 
to  other  on-line  systems  to  readiness  to  learn  the  DTIC  system. 

Part  II  seeks  specific  assessments  of  the  merits  of  the  training  materials 
and  equipment  and  of  the  physical  facility.  Part  III  also  seeks  direct  comment 
on  specific  aspects  of  the  intellectual  and  structural  components  of  the  course. 
Part  IV  offers  students  an  opportunity  to  provide  their  input  on  a  critical 
component,  the  performance  of  the  instructor.  Part  V  seeks  two  summary  measures 
of  the  success  or  failure  of  the  training  session.  While  gross  measures,  the 
information  these  summary  questions  elicit  should  be  useful  in  analyzing  trends 
over  time. 

Part  I,  Sections  A  and  B,  of  the  questionnaire  remained  unchanged  after  the 
pre-test.  In  the  first  version  of  Part  I,  C,  1  ("Other  On-line  Systems")  the 
respondent  was  asked  to  use  a  check-block  series  to  indicate  the  on-line  systems 
available  at  his  or  her  site.  The  respondent  was  then  asked  "which  ones  can  you 
operate?"  At  the  suggestion  of  several  pre-test  participants  the  Fame 
check-block  series  was  added  for  the  second  question.  Question  3  of  this  same 
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section  first  asked  if  knowledge  of  other  systems  helped  or  hindered  learning 
the  DTIC  system.  Several  participants  maintained  that  the  DTIC  system  is  so 
unlike  the  commercial  systems  that  the  category  "neither”  should  be  added  and  it 
was  added. 

Part  II,  B,  was  first  labeled  "Operation  of  equipment."  Some  participants 
were  confused  as  to  whether  this  question  asked  if  the  equipment  was  easy  to 
operate  or  if  it  would  operate  at  all.  As  a  consequence,  the  section  label  was 
changed  to  "Ease  of  equipment  operation." 

Section  C  of  Part  II  asks  the  respondent  to  evaluate  the  DROLS  "software 
functions  and  commands"  as  to  whether  they  are  “easy,  difficult,  or  need 
improvement."  Most  participants  marked  "difficult"  or  "need  improvement"  without 
further  comment.  However,  one  participant  observed  that  here  DTIC  was  asking  a 
question  to  which  it  already  knew  the  answer,  that  is,  that  DTIC  knows  that  its 
system  is  "horrendous  compared  to  other  systems." 

Part  III,  Section  H,  was  first  labeled  "Additional  training  needed." 

Several  participants  remarked  that  this  was  the  wrong  time  to  ask  this  question. 
They  agreed,  however,  that  if  the  label  was  changed  to  "Additional  training  you 
'relieve  you  will  need,"  it  would  then  be  an  acceptable  question.  The  suggested 
change  was  made. 

This  questionnaire  seeks  much  information  in  a  highly  structured  format. 

This  approach  should  provide  the  trainers  with  balanced  coverage  of  the  varied 
components  of  their  program.  It  will  also  provide  data  in  a  readily  analyzable 
form.  At  the  same  time,  by  means  of  several  "comments”  blocks,  the  student  is 
provided  with  a  number  of  opportunities  for  individualized  responses.  These 
responses  may  be  difficult  for  DTIC  personnel  to  categorize  but  should  provide 
valuable  insights  to  trainers. 

DTIC  REPORT  BIBLIOGRAPHY/MANAGEMENT  SUMMARY  INFORMATION  SYSTEM  QUESTIONNAIRE 


Upon  request  by  a  registered  user  for  information  on  a  particular  subject, 
DTIC  information  retrieval  analysts  will  conduct  a  tailor-made  computerized 
search  of  the  DTIC  data  bases.  These  data  bases  Include  the  Research  and 
Technology  Work  Unit  Information  System  (WUIS)  Data  Base,  the  Technical  Report 
Bibliographic  Data  Base,  and  the  Independent  Research  and  Development  (IR&D)  Data 


Base.  The  output  of  the  analyst's  search  Is  a  paper  document  called  a  "Report 
(Demand)  Bibliography,"  which  lists  relevant  reports  with  control  numbers, 
informative  abstracts,  and  data  describing  the  research  projects.  The 
bibliographies  are  free  of  charge  to  the  recipients. 

User  requests  are  submitted  by  mail  via  a  DTIC  form  which  seeks  the  subject 
and  scope  information  which  retrieval  analysts  need  to  construct  an  appropriate 
search  strategy.  Problems  will  arise  however  if  the  requestor  has  a  too  vague  or 
too  narrow  conceptualization  of  his  or  her  topic  or  of  how  that  tropic  relates  to 
the  contents  of  the  DTIC  data  bases.  Even  when  the  requestor  has  an  appropriate 
topic  conceptualization,  that  subject  statement  may  not  be  conveyed  adequately  to 
the  analyst  for  a  variety  of  reasons.  Communication  problems  are  sometimes 
exacerbated  because  Information  handling  specialists  often  prepare  the  DTIC 
request  for  the  end-user.  While  these  specialists  probably  clarify  many 
requests,  the  presence  of  an  intermediary  can  further  distort  the  communication 
between  the  requester  and  the  DTIC  analyst. 

This  questionnaire  (see  Figure  4)  was  developed  in  consultation  with  Fuller 
Murfree,  the  retrieval  analysts  of  the  Directorate  of  Data  Base  Services,  and 

Q 

Carol  Finney,  Office  of  User  Services,  Los  Angeles.  Some  of  the  retrieval 
analysts  were  enthusiastic  about  the  questionnaire,  seeing  the  information  it 
might  produce  as  an  aid  in  further  refining  their  skills.  Some  others  were 
negative  for  several  reasons.  Some  insisted  that  users  would  simply  refuse  to 
complete  the  questionnaires.  Others  seemed  to  believe  that  having  the  user  input 
would  have  no  practical  effect  on  agency  practices  or  procedures.  Another 
objection  appeared  to  center  around  a  suspicion  that  the  questionnaire  was  a 
management  device  to  place  blame  on  individual  analysts  for  user  dissatisfaction 
regardless  of  what  factors  may  have  contributed  to  an  unsuccessful  search 
product.  The  first  objection  can  be  addressed  by  proper  form  design  and 
aggressive  follow-up.  The  last  two  objections  can  not  be  addressed  by  survey 
methodology. 

Part  I,  "Description,"  is  Included  primarily  to  provide  retrieval  analysts 
with  problem  solving  Information  to  use  when  evaluations  are  unfavorable.  The 
first  question  is  a  gross  measure  of  familiarity  with  DTIC  procedures  and 
capacities.  The  second  asks  for  further  clarification  of  the  purpose  the 
requester  hoped  to  serve.  Question  3  seeks  clarification  of  the  subject  without 
the  intervention  of  an  intermediary.  Question  3  will  be  a  relatively  difficult 
one  and  may  not  produce  a  response. 


Figure  U 


I  DTIC  REPORT  BIBLIOGRAPH Y/MANAGMENT  INFORMATION  SYSTEM  SUMMAY  EVALUATION 


INSTRUCTIONS  TO  THE  USER:  The  objective  of  this  survey  is  to  obtain  information  for 
improvement  of  DTIC  Retrieval  Services.  The  evaluation  should  be  completed  by  the  PRINCIPAL 
USER  of  the  information.  Please  tell  us  what  you  like,  as  well  as  what  you  do  not  like, 
about  the  bibliography.  We  need  both  types  of  information.  Return  to  DTIC  within  five 


PART  I 

DESCRIPTION 

1 .  Was  this  your 

first  request  for  a  DTIC  bibliography? 

Yes 

_  No 

2.  What  did  you  expect  to  gain  from  this  search?  (check  one) 

_  Background  information  _  Other  (please  specify) 

_  Specific  information  _ 

Current  state-of-the-art  _ 


3.  SEARCH  STATEMENT;  Please  provide  a  summary  statement  of  the  subject  as  YOU 
conceptualized  it. 


5.  If  the  results  were  negative,  was  knowledge  of  this  beneficial?  _  Yes  _  No 


8.  Please  estimate  the  number  of  citations  which  were  NOT  RELEVANT  to  your  needs. 

_  5-10%  _  11-20%  _  21-30%  _  31-50%  _  51%  or  more. 


9.  What  is  your  overall  evaluation  of  this  search  on  the  following  scale?  (check  one) 


VERY 

USEFUL 


USEFUL 


OF  LITTLE 
USE 


OF  NO 
USE 


1  n 

n 

□ 

□ 

□ 

□ 

n 

x...... 

_ _ 

- 5 - 

- i - 

_ 1  -- 

_ I 

1 0 .  COMMENTS 


PART  III  APPLICATION 


11.  Was  the  search  product  beneficial  to  your  research  or  development  effort? 
Yes  No 


12.  If  the  answer  is  "yes,"  was  the  search  beneficial  in  the  saving  of  time? 

Please  estimate  amount  _ _ _ • 

I  (e.g..  Work-hours  saved  in  information  search  time,  work-hours  saved  in  avoidance  of 

work  duplication,  etc.) 


,  13.  If  the  answer  is  "yes,"  was  the  search  beneficial  in  the  savings  of  money? 

'  Please  estimate  amount  _ . 

I  (e.g.,  Dollar  value  of  savings  in  information  searc)i  time,  dollar  value  of  work  not 

duplicated,  etc.)  _ 


14.  PRINCIPAL  USER/EVALUATOR  NAME  AND  TITLE 


16.  Telephone  Numbers 

Autovon _ 

Commercial 


Date  survey  distributed 


Control 


Code 


Part  II,  Question  4,  seeks  to  measure  the  requester's  degree  of  satisfaction 
with  the  product  in  terms  of  its  timeliness.  In  many  situations  knowledge  that 
no  work  has  been  done  in  an  area  is  as  Important  to  the  researcher  as 
documentation  of  past  work.  Question  5  probes  for  this  situation.  Question  6 
may  produce  an  unfavorable  response  for  reasons  that  may  have  nothing  to  do  with 
the  quality  of  the  search  or  with  the  quality  of  the  data  base.  Nevertheless, 
the  number  of  citations  to  be  assessed  is  a  consideration  to  researchers  and  is  a 
frequently  expressed  source  of  dissatisfaction  to  users  of  information  products 
in  general.  Questions  7  and  8  seek  information  on  whether  the  user  believes  he 
or  she  has  received  an  acceptable  balance  between  comprehensive  coverage  and 
specificity,  that  is,  between  recall  and  relevance.  Question  9  is  again  a  gross 
measure  but  one  that  should  prove  useful  in  measuring  trends  over  time. 

Part  III,  "Application,"  Questions  12  and  13,  ask  the  respondent  to  place  an 
economic  value  on  the  information  product.  Answers  to  questions  such  as  this  are 
highly  speculative  at  best.  Many  users  will  see  such  questions  as  both  difficult 
and  futile  and  will  not  answer.  The  relatively  small  number  of  evaluators  who  do 
respond  however  will  provide  management  with  one  of  its  few  measures  of  program 
Impact.  While  such  figures  are  estimates  only,  they  can  be  useful  in  program 
advocacy  if  they  are  used  judiciously. 

Several  items  were  dropped  from  Part  I  as  the  result  of  the  pre-test.  The 
original  version  of  the  form  had  a  question  asking  if  the  user  had  provided  a 
narrative  statement  of  his  or  her  needs  and  another  asking  if  he  or  she  had 
provided  keywords  only.  The  questions  were  added  at  the  request  of  analysts  who 
wanted  to  emphasize  to  users  the  Importance  of  an  informative  narrative  statement 
of  user  needs.  These  questions  were  Included  even  though  the  designer  was  aware 
of  Che  precept  that  survey  instruments  should  only  attempt  to  elicit  information 
and  should  not  include  teaching  devices  disquised  as  questions.  The  questions 
were  dropped  when  pre-test  participants  quickly  observed  that  the  information  was 
already  available  to  UTIC  on  the  request  form. 

Question  8  was  added  at  the  suggestion  of  the  pre-test  participants.  They 
maintained  that  users  tend  to  note  Che  proportion  of  citations  which  are  not 
appropriate.  Most  participants  thought  that  few  persons  will  attempt  to  complete 
the  "Application"  section  and  that  those  who  do  will  be  giving  highly  speculative 
answers.  Wliile  this  assessment  is  probably  correct,  the  questions  were  retained 


for  the  reasons  stated  previously.  The  first  version  of  the  form,  in  the  "DTIC 
Use  Only"  section,  had  a  block  labled  “User  Code."  This  was  abbreviated  to 
"Code"  when  one  librarian/participant  said  that  for  control  purposes  she  would 
prefer  that  end-users  not  be  made  aware  of  the  oranizational  user  code. 

CURRENT  AWARENESS  BIBLIOGRAPHY  (CAB)  QUESTIONNAIRE 

The  Current  Awareness  Bibliography  (CAB)  Program  provides  a  customized 
automated  bibliographic  service  based  on  the  recurring  subject  needs  of  a  DTIC 
user.  The  user,  assisted  by  CAB  personnel,  selects  the  descriptors  and 
nonsubject  terms  such  as  COSATI  fields/groups,  corporate  source,  personal  author 
and  report  series  numbers  which,  taken  together,  comprise  the  "user  profile”  or 
strategy  used  to  search  the  data  base.  Every  two  weeks,  the  user's  subject 
Interest  profile  is  matched  against  Information  on  newly  acquired  documents.  The 
search  output  is  an  individualized  bibliography  in  paper  form  which  is  free  of 
charge  to  the  requestor.  The  number  of  user  search  profiles  varies  over  the  year 
but  averages  around  2,000.  These  2,000  profiles  are  for  persons  associated  with 
310  of  DTIC's  approximately  3,000  registered  user  organizations.^^ 

This  questionnaire  (see  Figure  5)  was  designed  in  consultation  with  Louis 
Williams,  Suzanne  Lynch,  and  Jane  Hatton,  Directorate  of  Data  Base 
Services. These  CAB  personnel  welcomed  the  prospect  of  obtaining  user 
feedback  on  their  service.  .  They  carefully  explained  the  operation  of  their 
program  so  that  the  questionnaire  could  be  most  effectively  focused  on  the 
realities  of  how  the  program  operates.  Serious  reservations  were  expressed, 
however,  about  prospects  for  achieving  a  satisfactory  return  rate  from  survey 
subjects. 

Part  I  of  the  questionnaire  seeks  to  determine  whether  users  are  keeping 
DTIC  informed  of  their  changing  needs.  The  first  two  questions  address  the 
adequacy  of  the  existing  search  profile.  Responses  to  Question  2  can  be 
categorized  to  identify  tendencies  to  mlsconstruct  profiles  according  to  certain 
patterns,  for  example,  to  construct  profiles  which  are  too  broad.  Number  3  seeks 
to  determine  whether  users  are  failing  to  notify  DTIC  when  they  are  no  longer 
satisfied  with  the  product. 

When  no  documents  related  to  an  interest  profile  are  input  to  the  DTIC 
system  during  the  two-week  cycle  the  person  for  whom  the  profile  was  constructed 


Figure  5 

CURRENT  AWARENESS  BIBLIOGRAPHY  (CAB)  EVALUATION  ~ 

I  INSTRUCTIONS  TO  THE  USER:  The  objective  of  this  survey  is  to  obtain  information  for 

improvement  of  the  service.  The  evaluation  should  be  completed  by  the  PRINCIPAL  USER  and 
|returned  to  OTIC  within  five  working  days. 

jpART  I  USER  SEARCH  STRATEGY  (SUBJECT  PROFILE) 

j  1.  Does  the  search  strategy  used  to  produce  your  CABs  adequately  represent  your  current 
I  areas  of  interest?  Yes  No 


!  2.  If  you  answered  "no"  above,  please  explain  why  the  search  strategy  is  not  appropriate 

I  and  how  you  think  it  could  be  improved. 

I 


3. 

Have  you  ever 

requested  that  your  search  strategy  be  revised? 

Yes 

_  No 

4. 

If 

no  citations  are  found,  is  notification  of  this  fact  beneficial  to  you? 
Yes  _  No 

5. 

Is 

the  format 

of  CAB  satisfactory?  Yes  No 

6. 

If 

the  format 

is  unsatisfactory,  how  would  you  like  to  have  it 

changed? 

Part 

II 

APPLICATION 

7.  How  will  you  use  this  CAB?  (Check  all  that  apply) 

A.  Keep  aware  of  current  DoD  research? 

P.  _  Find  citations  to  use  in  ordering  documents? 

C.  _  Other?  Please  specify  _ 


8. 

Does  any  other  engineer,  scientist,  or  research  person  regularly  use 
Yes  No  If  "yes",  how  many  others? 

your  CABS? 

- [ 

1 

I 

9. 

VJas  this  CAB  beneficial  to  your  research/development  effort? 

Yes 

— 

No  ! 

1 

1 

10  . 

If  the  answer  is  "yes,"  was  the  CAB  beneficial  in  savings  of  time? 
Please  estimate  amount: 

(e.g.,  work-hours  saved  in  information  search  time,  work-hours  saved 
work  duplication.) 

in 

r 

avoidance  of 

1 1  . 

If  the  answer  is  "yes,"  was  the  CAE  beneficial  in  savings  of  money? 
Please  estimate  amount: 

(e.g.,  dollar  value  of  savings  in  information  search  time,  dollar  value 
duplicated,  etc.). 

of  work 

not 

PART 

Ill  USER  INFORMATION 

:  12.  PRINCIPAL  USEK'J 

:  NAME  AMJ  TITLE 

13. 

Date  of  Evaluation 

1 

i 

14. 

Telephone  Numbers 

Autovon 

Commercial 

Date  survey  distributed 


Code 


Control  number 


is  notified  of  that  fact.  Question  4  seeks  to  determine  whether  this 
notification  of  negative  search  results  actually  is  of  value  to  users.  Questions 
5  and  6  are  straightforward  requests  for  format  evaluations  with  requests  for 
ideas  for  change. 

Part  11,  Questions  7  and  8  try  to  determine  how  the  CABs  are  actually  being 
used  and  if  they  are  serving  a  greater  number  of  persons  than  DTIC 's  official 
user  tally  would  Indicate.  Questions  9,  10,  and  11  are  identical  to  ones  on  the 
Demand  Bibliography  Evaluation  form  and  are  intended  to  serve  those  same 
purposes. 

In  the  first  version  of  the  form  the  writer  included  the  question,  "If  you 
wanted  to  have  your  search  strategy  revised,  what  DTIC  office  or  person  would  you 
call?"  Pre-test  participant  responses  Indicated  that  recipients  do  not  in  fact 
know  who  to  call  and  that  they  think  this  is  a  detail  they  should  not  be  expected 
to  know.  The  response  to  the  question  was  sufficiently  negative  to  warrant  its 
being  dropped. 

Two  of  the  pre-test  participants  predicted  that  problems  will  arise  because 
the  form  is  addressed  to  the  end-user  while  the  librarian,  as  intermediary,  plays 
an  important  part  in  the  process.  For  example,  one  stated  that  she  routinely 
removes  the  search  strategy  listing  (which  accompanies  the  bibliographies) 
before  she  passes  the  CABs  on  to  the  end-user. 

Questions  9  through  11  produced  the  same  caveats  with  respect  to  CAB  as  they 
did  with  respect  to  Demand  Bibliographies.  The  questions  were  retained,  despite 
their  drawbacks,  for  the  same  reasons  they  were  retained  on  the  Demand 
Bibliography  evaluation  form. 

RECUKKING  MANAGEMENT  INFORMATION  SYSTEM  REPORTS  QUESTIONNAIRE 

These  reports,  focusing  on  research  in  progress,  are  compilations  of  data 
from  the  Work  Unit  Information  System  and  the  Independent  Research  and 
Development  data  bases.  They  may  be  requested  on  an  automated  recurring  basis, 
to  be  complied  monthly,  quarterly,  semiannually,  or  annually.  A  search  strategy 
or  profile  of  the  user’s  area  of  interest  is  developed  through  consultation 
between  a  DTIC  representative  and  the  requester.  The  output  format  is  a  paper 
document  which  is  free  of  charge  to  the  recipient.  The  number  of  Recurring 


Report  profiles  as  of  13  May  1983  stood  at  1,247  for  persons  within  221 
organizations. 

This  evaluation  form  (see  Figure  6)  was  designed  in  consultation  with  Roland 
Russell,  Directorate  of  Data  Base  Services.  Like  the  CAB  personnel,  Mr. 

Russell  took  a  very  positive  approach  toward  the  design  of  a  user  feedback 
mechanism.  He  also  shared  their  reservations  about  how  successful  such  a  program 
could  be  without  a  well-developed  system  to  deal  with  non-response. 

The  Recurring  Reports  program  has  many  structural  and  procedural 
similarities  to  the  CAB  program  and,  as  a  consequence,  much  the  same  questions 
can  be  used  for  both  evaluation  forms.  Questions  1-4  are  identical  to  Questions 
1-4  on  the  GAB  form  and  are  intended  to  serve  the  same  purposes  as  explained  in 
the  previous  section.  Part  II,  "Application,"  is  parallel  to  Part  II  of  the  CAB 
questionnarire.  The  purposes  are  the  same  in  both  cases.  Unlike  the  CAB 
questionnaire,  however,  this  form  has  no  questions  regarding  format  because  of 
proprietary  restrictions  governing  who  may  obtain  what  information  in  a  Recurring 
Report . 

Pre-test  participant  response  to  the  Recurring  Reports  questionnaire  was 
substantially  the  same  as  the  response  to  the  CAB  questionnaire.  Two 
participants  suggested  that  a  check-block,  "Identify  colleagues  working  in  the 
same  research  area,"  be  added  to  Part  II,  5.  They  maintained  that  this 
capability  is  one  of  the  most  important  aspects  of  the  Recurring  Reports  service. 
This  addition  was  made  to  the  questionnaire. 

AUTOMATIC  DOCUMENT  DISTRIBUTION  (ADD)  PROGRAM  QUESTIONNAIRE 

This  program  provides  for  automatic  distribution  of  microfiche  copies  of 
technical  reports  selected  according  to  the  user's  area  of  interest.  The  service 
anticipates  the  user's  needs  by  means  of  a  computerized  comparison  of  the  user's 
subject  profile  against  a  data  base  of  newly  accessioned  technical  reports.  The 
user  profiles  are  developed  according  to  the  same  principles  as  CAB  and  Recurring 
Reports  profiles.  The  mcrofiche  sets,  which  are  shipped  every  two  weeks  at  a 
cost  of  !?.35  per  microfiche  report,  are  available  to  DTIC  users  having  a  deposit 
account  with  the  National  Technical  Information  Service,  The  ADD  program 
presently  consists  of  241  profiles  for  recipients  at  198  user  organizations.  ADD 


Figure  6 


RECURRING  MANAGEMENT  INFORMATION  SYSTEMS  (MIS)  REPORTS 


INSTRUCTIONS  TO  THE  USER:  The  objective  of  this  survey  is  to  obtain  information  for 
improvement  of  the  service.  This  evaluation  should  be  completed  by  the  PRINCIPLE  USER  of 
of  the  reports  and  returned  to  OTIC  within  five  working  days. 


PART  I  USER  SEARCH  STRATEGY  (SUBJECT  PROFILE) 


1 .  Does  the  search  strategy  used  to  produce  your  Recurring  MIS  Reports  adequately 
represent  your  current  areas  of  interest?  Yes  No 


2.  If  you  answered  "no"  above,  please  explain  why  the  search  strategy  is  not  appropriate 
and  how  your  think  it  could  be  improved. 


[I  APPLICATION 


How  will  you  use  this  Recurring  MIS  Report?  (check  all  that  apply) 

A.  _  Keep  aware  of  current  DoD  research 

B.  _  Identify  colleagues  working  in  the  same  research  area 

C.  _  Other.  Please  specify  _ . 


Does  any  other  engineer,  scientist,  or  research  person  regularly  use  your  Recurring 
MIS  Reports?  _  Yes  _  No  If  "yes,"  how  many  others?  _ . 


Was  this  Recurring  MIS  Report  beneficial  to  your  research/development  effort? 

_  Yes  _  No 


If  the  answer  is  "yes,"  was  the  Recurring  MIS  Report  beneficial  in  savings  of  time? 

Please  estimate  amount  _ _ _ . 

(e.g..  Work-hours  saved  in  information  search  time,  work-hours  saved  in  avoidance  of 
of  work  duplication.) 


If  the  answer  is  "yes,  was  the  Recurring  MIS  Report  beneficial  in  savings  of  money? 

Please  estimate  amount  _ 

(e.g..  Dollar  value  of  savings  in  information  search  time,  dollar  value  of  work  not 
duplicated,  etc.). 


PRINCIPLE  USER  IDENTIFICATION 


PRINCIPAL  USER'S  NAME  AND  TITLE 


1 1 .  Date  of  Evaluation 


12.  Telephone  Numbers 

Autovon  _ 

Commercial 


USE  ONLY;  Date  survey  distributed  Control  number  Code 


program  microfiche  sets  are  generally  handled  and  retained  in  organization 
information  facilities. 

The  evaluation  form  (see  Figure  7)  for  the  ADD  program  was  designed  in 
consultation  with  Timothy  McCleery,  Directorate  of  Document  Services,  and  with 
the  members  of  the  ADD  Quality  Circle. Because  claims  for  missing 
shipments  are  a  continuing  concern,  ADD  personnel  were  particularly  Interested  in 
receiving  input  on  how  their  product  is  processed  at  the  local  level. 
Additionally,  they  warned  that  problems  may  be  encountered  in  administering  the 
questionnaire  because  certain  organizations  have  multiple  search  profiles  and  in 
some  few  cases  multiple  user  codes,  e.g. ,  military  schools  with  a  separate  user 
code  for  their  theses. 

Fart  I,  Questions  1,  2  and  3,  address  the  user  search  strategy  in  much  the 
same  way  and  for  much  the  same  reasons  as  did  the  CAB  and  Recurring  Reports 
questionnaires.  Because  the  ADD  product  is  a  document  per  se  rather  than  a 
surrogate  there  is  more  concem  for  the  specifics  of  how  the  product  is  handled 
at  the  local  level.  Question  5  seeks  the  recipient's  assessment  of  the  effect 
microfiche  have  on  their  ordering  of  paper  copies  of  the  same  documents.  This  is 
a  matter  of  interest  since  opinion  is  divided  as  to  whether  the  examination  of 
microfiche  copies  actually  stimulates  the  end-user  to  seek,  paper  copies  or 
whether  microfiche  are  accepted  as  substitutes.  Questions  6  and  7  seek  to 
determine  to  what  extent  automatic  distribution  of  information  items  actually 
saves  time  for  professional  information  handlers.  Claims  for  missing  shipments 
are  a  continuing  concern  to  DTIC.  Question  8  seeks  to  determine  the  extent  to 
which  users  verify  shipments  at  the  time  of  receipt.  It  is  uncertain  how 
satisfactory  the  present  billing  system  actually  is  to  users  and  Question  9  is  a 
straightforward  attempt  to  shed  light  on  this  question. 

Question  3  of  the  first  version  of  the  questionnaire  asked  if  the  evaluator 
had  requested  a  search  strategy  change  within  the  past  one,  two,  or  three  years. 
At  the  suggestion  of  pre-test  participants,  the  time  periods  were  changed  to  six 
months  and  one  and  two  years.  The  first  version  also  asked  if  the  evaluator  knew 
which  DTIC  offices  to  call  to  change  a  search  strategy  or  to  resolve  problems 
with  billing.  These  two  questions  were  removed  when  the  pre-test  participants 
indicated  that  they  did  not  know  and  saw  no  reason  why  they  should  know  the 
answers  to  these  questions. 
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ADD  PROGRAM  EVALUATION 


L' 


P 


INSTRUCTIONS  TO  THE  USER:  The  objective  of  this  survey  is  to  obtain  information  to  improve 
the  service.  The  evaluation  should  be  made  by  the  person  most  knowledgeable  about  how  the 
ADD  program  is  used  in  your  information  facility.  Your  cooperation  and  frankness  will  be 

appreciated.  Please  return  to  DTIC  within  five  working  days. _ 

PART  I _ USER  SEARCH  STRATEGY  (SUBJECT  PROFILE) _ 


1.  Does  the  search  strategy  (subject  profile)  used  to  produce  your  ADD  package  adequately 
represent  your  current  areas  of  interest?  Yes  _  No 


2.  If  you  answered  "no"  to  the  above,  please  describe  the  ways  in  which  the  search  strategy 
produces  inappropriate  results. 


3.  When  did  you  last  request  that  your  search  strategy  be  revised? 

_  Within  the  past  6  months?  _  Within  the  past  year?  _  Within  the  past  2  years? 


PART  II _ USAGE  OF  ADD  DOCUMENTS _ 

4.  Please  estimate  the  percentage  of  your  ADD  documents  which  are 

a.  _  %  routed  directly  to  individual  researchers. 

b.  _  %  added  to  permanent  library  collection. 

c.  _  %  discarded  as  unrelated  to  your  area  of  interest. 

id.  %  other 


5.  What  effect  do  the  ADD  Program  fiche  have  on  your  ordering  of  paper  copies? 

(check  one)  _  Encourage  ordering  of  paper  copies  _  Substitute  for  paper  copies 

No  effect  noticed 


6.  Does  the  ADD  Program  save  you  time?  Yes  _  No 


7.  If"  yes,"  in  which  activities  does  it  save  time?  (check  all  that  apply) 
_  Selection  _  Ordering  _  Other  (Please  specify)  _ 


PART  III _ PROBLEM  AREAS _ 

8.  When  an  ADD  shipment  arrives  at  your  library,  do  you 

_  check  the  shipment  against  the  packing  list. 

_  selectively  check  the  shipment  against  the  packing  list. 

_  not  check,  relying  on  DTIC  accuracy. 


9.  Do  you  have  problems  with  billing  for  the  ADD  Program?  Yes  No 

PART  IV  EVALUATOR  IDENTIFICATION 

10.  EVALUATOR  NAME  AND  TITLE: 

11.  Evaluator  Telephone  Number (s) 

Autovon 

Commercial  (  ) 

DTIC  USE  ONLY:  Date  survey  distributed 


Control  number 


Code 


USER  CONCERNS  ANALYSIS  GRID 


This  format  (see  Figure  B)  Is  suggested  as  an  aid  for  DTIC  personnel  to  use 
In  Identifying,  categorizing  and  analyzing  the  concerns  that  users  express  at 
conferences.  Conference  minutes  should  be  examined  for  expressions  such  as  "why 
can't  DTIC.  .  . , "  "It  would  help  If.  .  "you  never  have.  .  etc.  Once  the 

user  concerns  are  Identified,  they  should  be  categorized  according  to  the 
following  criteria: 

1.  Importance.  The  concern  that  users  seem  to  feel  about  the  problem.  For 
example,  do  they  seem  to  believe  that  the  matter  Is  critical  and  do  they 
seem  willing  to  exert  pressure  to  Influence  the  agency  to  find  a  solution? 

2.  Resources  needed.  The  financial  and  human  resources  that  would  be 
required  to  achieve  a  solution.  For  example,  would  the  solution  require 
outlay  of  monies  considerably  beyond  current  budget  projections. 

3.  Authority.  The  level  at  which  the  decision  can  be  made  to  Implement  a 
solution  to  this  problem.  Is  the  solution  one  that  can  be  Implemented 
without  authorization  from  DLA  or  DoD? 

4.  Complexity.  Is  this  a  problem  to  which  the  solution  Is  clear  to 
everyone  concerned  or  Is  this  a  problem  to  which  the  solution  Is  not 
apparent  and  further  analysis  would  be  required? 

5.  Time.  Can  the  problem  be  solved  in  a  short  time?  Or  is  it  something 

that  would  require  a  long  time  owing  to  personnel,  budgetary,  organizational 

or  technological  constraints? 

6.  Results.  Will  the  agency  and  the  users  know  when  the  problem  is  solved? 

Or  has  the  problem  been  conceptualized  In  such  a  way  that  there  will  be 

perpetual  cries  for  "more  and  better."  If  the  latter  is  the  case,  then  the 
problem  should  be  restated  so  that  the  results  can  be  measured. 

The  grid  shows  the  weighting  scheme  for  prioritizing  concerns.  A  format 
such  as  this  will  help  make  the  decision-making  process  more  orderly. 
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Elizabeth  Quinn 

US  Army  Concepts  Analysis  Agency 
Bethesda,  MD  20814 
202-295-1530 
2  May  1983 


J.  Marshal  Hughes,  II 
Dahlgren  Laboratory 
Naval  Surface  Weapons  Center 
Dahlgren,  VA  22448 
703-663-8994 
3  May  1983 


Patricia  Pulliam 

Dahlgren  Laboratory 

Naval  Surface  Weapons  Center 

Dahlgren,  VA  22448 

703-663-8994 

3  May  1983 


Mary  Jane  Brewster 

Naval  Surface  Weapons  Center — Whiteoak 
Silver  Spring,  MD  20910 
301-394-3550 
3  May  1983 


Joanne  Lappin 

David  W.  Taylor  Naval  Ship  Research  and  Development  Center 
Bethesda,  MD  20084 
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4  May  1983 


Ruth  Smith 

National  Technical  Information  Center 
Springfield,  VA  22151 
703-487-4696 
5  May  1983 
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Sami  Klein 

Library  and  Information  Services  Division 
National  Bureau  of  Standards 
Gaithersburg,  MD  20460 
301-382-5927 
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Betsy  Fox 

Defense  Nuclear  Agency 
Alexandria,  VA  20305 
202-325-7780 
6  May  1983 
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views,  12,  19,  and  29  April  and  10  May  1983. 

Murfree,  Fuller,  et.  al.  Defense  Technical  Information  Center,  Alexandria,  VA. 
Interviews,  9,  25,  27  and  29  April  1983. 

Pickeral,  Judy.  Defense  Technical  Information  Center,  Alexandria,  VA.  Inter¬ 
view,  6  May  1983. 

Russell,  Roland.  Defense  Technical  Information  Center,  Alexandria,  Va.  Inter¬ 
views,  26  and  27  April  and  10  and  13  May  1983. 

Williams,  Louis  ,  Suzanne  Lynch  and  Jane  Hatton.  Defense  Technical  Information 
Center,  Alexandria,  VA.  Interviews,  20  and  26  April  and  10  and  13  May 


